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Abstract 

Airflow hazards such as vortices or low level wind shear have been identified as a primary 
contributing factor in many helicopter accidents. US Navy ships generate airwakes over their decks, 
creating potentially hazardous conditions for shipboard rotorcraft launch and recovery. Recent sensor 
developments may enable the delivery of airwake data to the cockpit, where visualizing the hazard data 
may improve safety and possibly extend ship/helicopter operational envelopes. A prototype flight-deck 
airflow hazard visualization system was implemented on a high-fidelity rotorcraft flight dynamics 
simulator. Experienced helicopter pilots, including pilots from all five branches of the military, 
participated in a usability study of the system. Data was collected both objectively from the simulator 
and subjectively from post-test questionnaires. Results of the data analysis are presented, demonstrating 
a reduction in crash rate and other trends that illustrate the potential of airflow hazard visualization to 
improve flight safety. 


Introduction 

The dangers that airflow hazards pose to helicopter 
pilots may be mitigated by new hardware developments 
that can provide airflow data to the cockpit. The challenge 
then becomes how to concisely present this large volume of 
data to the pilot. We discuss the process of user-centered 
design by which a prototype of an airflow hazard 
visualization system was developed. W r e describe the 
system implementation, and the protocol, methodology and 
results of a flight simulation usability study. The presence 
of the visual system dramatically reduced the crash rate for 
helicopters flying into simulated hazardous conditions. 

Airflow Hazards 

Turbulence and other wind-related factors were 
implicated in nearly 10% of the over 21,000 aircraft 
accidents in the US National Transportation Safety Board 
accident database from 1989-99 [1], Encounters with 
airflow hazards such as vortices, downdrafts, low level 
wind shear, microbursts, or turbulence from surrounding 
vegetation or structures near the landing site can be deadly 
to aircraft of all categories and classes. However, 
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helicopters are especially vulnerable to airflow' hazards 
because they often have to operate in confined spaces and 
under operationally stressful conditions (such as 
emergency search and rescue, military or shipboard 
operations). 

Airflow hazards are hard to detect simply because air is 
invisible. Disturbed airflow is undetectable by pilots on a 
landing approach unless the air happens to pick up dust, 
smoke or other aerosols that are visible to the human eye. 
Being thus unable to directly see a factor of potentially 
great importance to them, pilots learn to use their intuition 
to predict airflow patterns over obstacles near the takeoff 
or landing site, and they learn to pick up visual cues from 
the surrounding area. However, airflow-related accidents 
still occur. 

Helicopter Shipboard Operations 

Operating a helicopter from a moving aircraft carrier is 
one of the most demanding tasks a helicopter pilot can face 
[2, 3], The pilot must maneuver the helicopter within very 
tight tolerances to avoid striking ship structures or other 
aircraft. In addition, high sea states may cause extreme 
deck angles of pitch and roll, and low visibility may 
degrade visual cues. Furthermore, because the ship is 
moving, its superstructure will ahvays generate an airwake 
consisting of vortices and other airflow hazards, adding to 
the challenge of shipboard launch and recovery. It is a task 


that demands the utmost concentration and skill from the 
pilot. A system that can deliver even an incremental 
amount of assistance to the pilot could yield a significant 
safety improvement in this domain. 

Helicopter accidents and incidents that occur on 
shipboard range from incidents such as tunnel strikes to 
fatal accidents. Over 120 tunnel strikes have occurred in 
dual-rotor helicopters since the 1960s, causing damage 
ranging from S50K. to over SIM per incident [4]. Analysis 
of these accidents and incidents frequently finds airflow 
hazards to be the root cause. The pilot and ground crew 
are initially unaware of the danger, and the pilot is unable 
to react in time. Presenting the appropriate information that 
could enable the flight crew to make correct decisions in 
advance of the hazard encounter, therefore, could reduce or 
prevent such accidents. 

Because shipboard rotorcraft operations are such a 
demanding environment, the US Navy’s Dynamic Interface 
flight test program has compiled significant amounts of 
data from shipboard flight tests, wind tunnel tests, and 
computational fluid dynamics computations studying the 
aerodynamics of shipboard-rotorcraft interactions [5]. 

Conveying Ship Airwake Information to Pilots 

The current method of communicating this information 
to the pilots consists of publishing pre-computed 
operational envelopes (Figure 1) listing allowable wind 
conditions for many ship-rotorcraft combinations [6, 7], 
The envelope conveys a go/no-go decision, and does not 
state which safety considerations motivate a given 
operational limit. Pilots check the published envelope for 
their helicopter before beginning any approach, and they 
fly the approach only if they are within the envelope. This 
procedure has the advantage of providing clear, simple 
direction to the pilots under all wind conditions. However, 
if the winds shift out of the envelope during the approach, 
or some other event occurs that changes the airflow over 
the landing site, such as a helicopter on an upwind spot 
starting up its rotor, a hazardous condition can occur of 
which the pilot is unaware. This type of situation has been 
demonstrated to be a causal factor in many accidents and 
incidents [8], 



Figure 1. Shipboard rotorcraft operational envelope 


New Sensor Technology 7 

Recent advances in sensor technology such as Doppler 
lidar [9] and other techniques axe leading to the 
development of aircraft-based sensors which can collect 
large amounts of airflow velocity data in real time. It is 
likely that aircraft-mounted hardware will soon be 
available that can reliably scan the area a few hundred feet 
ahead of the aircraft and sample air particle vector 
velocities at one-foot intervals or less [10]. With the 
development of such devices, onboard detection systems 
that can convey detailed information about airflow hazards 
to pilots in real time become a possibility. Such systems 
will require an interface that can concisely present large 
amounts of data to the pilot in a comprehensive manner in 
real time, yet not distract from the pilot’s primary task of 
flying the aircraft. 

Airflow Hazard Visualization 

Initial Usability’ Study 

In a preliminary usability study [11], we presented 
numerous visual representations of regions of hazardous 
airflow to pilots, while simulating the cockpit view of a 
helicopter’s final approach to shipboard landing on a 
projection screen. Variables studied included shape, color, 
and animation of the hazard indicators. 


Common techniques used by flight test engineers in 
understanding ship airwake usually include 3D motion, 
such as smoke trails injected into wind tunnels. Viewers of 
the video sequences often find the visualization of the air 
particles more instructive than static presentations [6], 
However, upon being shown animated imagery over 
shipboard landing sites, the pilots in our preliminary study 
strongly rejected the use of dynamic indicators. 

The pilots favored much simpler imagery than we had 
initially anticipated. Helicopter pilots landing on shipboard 
must focus all their attention to . complete the landing 
safely, and have little spare cognition to analyze detailed 
quantitative information about hazards. An abundance of 
detail, motion (animation), complex shapes, and too many 
colors were all ruled out as distracting and possibly 
dangerous in the high-demand environment. The visual 
indicators had to be sufficiently translucent so as not to 
obscure any critical shipboard visual cues that the pilots 
needed as landing aids. The pilots desired to be informed 
only of the location of the hazard and its severity — a 
warning (yellow) or danger (red). In other words, our 
domain experts had informed us of the need for a decision 
support system with minimum critical information, not a 
scientific visualization system, and their reasons had to do 
with the division of attention in the high-demand 
environment. 

This first phase of the study also revealed a strong 
preference by the pilots for a display in which the hazard 
indicator appears to be spatially conformal with the actual 
hazard in the physical scene. During potentially dangerous 
conditions, the pilot’s attention will inevitably be focused 
outside the cockpit during the critical landing moments; he 
or she will not want to dance away and down at a cockpit 
instrument display. The pilots strongly favored an 

augmented-reality hazard visualization display on a head- 
up display (HUD). However, the display must be 

thoughtfully designed not to distract from the key 
shipboard visual cues, especially when these cues are 
degraded during a nighttime or poor-weather landing. 

Earlier studies have demonstrated that head-up displays 
with superimposed symbology may occasionally cause 
performance problems due to attentional capture by the 
perceptual grouping of the superimposed symbols [12, 13], 
“Scene-linked” head-up displays, or displays where there is 
no differential motion between the superimposed 
symbology and the outside scene, can solve this problem. 

Our study also confirmed the requirement for a head-up 
display where the hazard indicator is three-dimensional and 
appears to be physically part of the world. 

The pilots stressed the importance of utilizing 
conventional symbology at all times. They emphasized the 
danger even a moment of confusion could cause, and 
strongly recommended that the symbology used in our 


head-up display conform to current aviation standards. It 
was particularly important that our symbols not have any 
chance of being confounded with other types of HUD 
symbology already in use. The results from this prototype 
study enabled us to select a design that was substantially 
different from any existing type of HUD symbology. 

Implementation of Flight Simulation Interface 

With the knowledge gained from the results of the 
preliminary study, we implemented a version of our 
interface in Advanced Rotorcraft Technology’s (ART) 
high-fidelity rotorcraft simulator [14], a fixed-base, 
aerodvnamically accurate flight simulator with a three 
projection screen display (Figure 2). ART, located in 
Mountain View, California, is a rotorcraft flight simulation 
company specializing in non-linear dynamics modeling and 
analysis [14], 



Figure 2. ART flight simulator with pilot in front of 
projection screen and operator at rear console 


ART’S visual subsystem is layered on top of OpenGVS 
[15], an OpenGL-based [16] scene manager built by 
Quantum3D. As a result, we could generate complex 
three-dimensional OpenFlight [17] objects in MultiGen 
software, import them into ART’s flight simulator graphics 
subsystem, and manipulate them as desired in the flight 
simulator scene. OpenGL is an industry-standard API for 
developing 2D and 3D graphics applications. OpenFlight 
is a commercial, hierarchical 3D scene description file 
format, based on OpenGL. 

Simulator Validation and Quality 

ART's aerodynamic models have been verified by the 
US Navy via stability and control techniques and frequency 
domain validation [18, 19],- and Navy flight test engineers 
and pilots have stated that they are more aerodynamically 
accurate than other rotorcraft flight simulators currently 
available [18], 

The only formal criteria to validate the performance of 
a high fidelity rotorcraft dynamic flight model are those in 
FAA Advisory Circular 120-63, Helicopter Simulator 


Qualification [20]. ART’s dynamic models do not folly 
meet the FAA Level D specifications, although they are 
very close in many flight regimes. However, these criteria 
are intended for training simulations (for example, the 
aircraft cockpit must be faithfully depicted) and are not as 
relevant for our purposes since we do not need to train 
helicopter pilots, but instead are looking for an 
aerodynamically accurate flight simulation. Additionally, 
the criteria are so difficult for rotorcraft simulators to meet 
(the error tolerance in measured rotorcraft data is often 
greater than the Level D specifications; for example, Level 
D requires that the torque error is within 3%, which also 
falls within the modem flight test measurement error range 
[18]), that there are no physics-based rotorcraft flight 
models available today that folly satisfy the FAA Level D 
requirements for rotorcraft [19]. 

Simulator Specifications 

The study was performed in a high fidelity helicopter 
flight dynamics simulator with a single seat configuration, 
flight controls with force feedback, instrument panel, and a 
three-channel projection outside world visual system 
utilizing 3D Perception projectors to provide 1024 x 768 
resolution at 1000 ANSI lumens. Visual rendering is done 
using ART software that supports rendering on OpenGL 
graphics cards using OpenFlight format visual databases. 
Image generation is done on PCs with graphic acceleration 
hardware that provides a 60 Hz update rate with foil-screen 
anti-aliasing and a 188” horizontal by 54” vertical field of 
view on a 6.5-ft radius cylindrical screen. 

An operator console provides foil simulator control, 
monitoring of the visual system and instrumentation 
displays, initialization to saved reset points and arbitrary 
test conditions. Control loaders for the pilot’s controls are 
electric and are driven by software that interfaces the flight 
dynamics model to the control loaders and edits the force 
feel characteristics. Four sets of control loaders are used to 
drive the longitudinal cyclic, lateral cyclic, collective and 
pedal controls. Computer generated images are rendered of 
the instrument panel. A dual 1.9GFIz AMD processor 
computer with two graphics boards, located in the operator 
console, is used to drive a flat panel display that is mounted 
behind instrument panel overlays. 

Flight Simulation Usability Study 

Study Design and Implementation Process 

We used a three-phase iterative design process in 
developing the interface and the study protocol. Highly 
detailed and realistic 3D models of a Sikorsky UH-60 
Seahawk helicopter (Figure 3) and a Navy LHA (Tarawa- 
class) ship (Figure 4) had already been input into the flight 
simulator system. Additional details of the simulation 
study can be found in [21] and [22]. 



Figure 4. LHA steeming, losded with sircrsft (US 
Govt, image) 


In the first phase, we reviewed US Navy Dynamic 
Interface (DI) helicopter-shipboard flight test data for the 
H-60 and LHA and selected four critical scenarios where, 
depending on the speed and direction of the wind over the 
ship deck, hazardous airflow could occur (Figure 5). 





Figure 5. Photo from ART flight simulator, labeled: 
landing spots and hazard locations for the four 
scenarios on the LHA-1 ship 


conditions that would create approaches with varying 
landing difficulties for each ship/wind-direction scenario. 
An experienced Navy helicopter test pilot flew all the 
stored approaches to verify the realism of the simulation, 
the location of the areas of hazardous airflow, and the 
validity of our landing difficulty ratings. 

For the second phase, based on the test pilot’s input 
and after lengthy examination of the airflow data, we 
created translucent 3D OpenFlight surfaces that outlined 
the volumetric regions of hazardous flow (Figure 6). 
(Actual surfaces were jnore translucent than pictured in the 
figure.) Based on the results from the study of the low- 
fidelity prototype, we had selected a simple, static design 
for the hazard indicators and used only two colors, yellow 
(caution) and red (danger). The shape and appearance of 
the indicators were chosen to indicate the physical location 
of the hazard without undue distraction and without 
duplicating any symbology used for other purposes, while 
the color meanings are conventional and widely accepted 
in the aviation world. 


We defined a “scenario” as a combination of wind 
direction and approach to a landing spot (the LHA had ten 
different landing spots) where hazardous airflow could 
occur near or over the chosen landing spot (Table 1). In 
situations similar to these, accidents had occurred in the 
past. 


Table 1. Flight Simulator Scenario Descriptions 

Simulator Scenario Descriptions 1 
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Notes: 

1. Ship, Course, Speed, Sea State, Temperature, Ship Motion, Gross Weight, 
Loading are identical for all scenarios. Ship Course is directly to North or 360 
degrees. 



Figure 6. Visual hazard indicators used in the study 
for the four scenarios, Aft, Bow, Port, and Starboard 
(actual indicators were more translucent than 
depicted) 

The objects were imported into the simulator’s visual 
subsystem, scaling, rotating, and translating them into their 
proper positions on the LHA. This was done manually in 
order to accurately correlate the surfaces with the known 
areas of hazardous airflow from our study of the data. The 
objects were linked to the ship so that they seemed to be 
part of the simulated outside world; they appeared as 
clouds or curtains hovering over particular locations on 
shipboard. This is an accurate model of shipboard airwake; 
any hazardous areas produced by wind blowing over ship 
structures will move along with the ship. 


We input actual airflow data from Navy DI flight tests, 
computational fluid dynamics (CFD) calculations, and 
wind tunnel tests into the simulator. We estimated wind 


The following figures (Figure 7)(Figure 8)(Figure 
9)(Figure 1 0) are digital photos taken in the simulator room 
at Advanced Rotorcraft Technology, Inc. (ART) that depict 
the visual appearance of each of the four hazard indicators 
for each of the Aft, Bow, Port, and Starboard scenarios. 





Figure 7. Hazard indicator (yellow, caution) - Aft 


scenario 



Finally, an experienced Navy test pilot flew all the 
approaches and performed a final verification of the correct 
placement of the hazard indicators as well as the validity of 
the stated difficulty levels of the approach. 

At this point, we were confident that we had a set of 
* realistic, aerodynamically accurate approaches for 
helicopter pilots landing on an LHA ship. We 
checkpointed all 28 different approaches, plus four practice 
approaches with light winds for the orientation flight, over 
four scenarios in preparation for our flight simulation 
usability study. 

Study Frotocoi and Design 

The study was a 3 (landing difficulty) x 2 (presence or 
absence of visual hazard indicator) x 4 (approach type) + 1 
x 1 x 4 (control) within- subjects design. Each pilot flew the 
same 28 simulated approaches, but in different orders. 

Each participant received a pre-flight briefing that 
explained the structure of the simulation and the use of the 
controls of the simulator and instructions as to the meaning 
of the yellow and red hazard indicators. Participants then 
performed a series of orientation flights before beginning 
the actual test. There were five orientation flight 
sequences. First, pilots were given a few minutes to 
accustom themselves to the “feel” of the simulator by 
flying the simulated helicopter from a low speed up to 
cruise and back down to a hover, and then flying around 
the ship and simulated terrain. Then the pilot flew four 
approaches, one to each of the four targeted landing spots 
for the test scenarios, but with low (non-hazardous) winds. 
Thus they were familiarized with the environment and the 
out-the-cockpit view for each of the approach scenarios. 



The dual purposes of the orientation flights were to 
accustom them to the feel of the controls of the simulator, 
and to determine if they had the skill level to be a credible 
participant in the experiment. Out of 17 pilots recruited for 
the study, one was unable to fly the orientation flights and 
was excused, leaving 16 pilots who then completed the test 
approaches. 

At the outset of each approach, pilots were given wind 
direction but not wind speed. Revealing wind speed could 
introduce bias due to the pilots’ assumption that wind 
speed correlates with landing difficulty level, although 
pilots were briefed that hazards could occur even at low 
wind speeds. 

Participants 

We recruited 17 military and civilian helicopter pilots 
by word-of-mouth and through emailed requests for 
volunteers. 1 6 pilots ( 1 female) flew the orientation flights 
successfully and completed the simulation test. This group 
of pilots had no previous experience on the simulator used 
in the experiment and had not seen or heard of any type of 
visual hazard indicating system before. Pilot experience 
ranged from 200 to 7300 helicopter flight hours with the 
median number of hours being 2250, from 2 to 46 years of 
experience as a helicopter pilot with the median 13 years, 
and were from 25 to 65 years old, with a median age of 36 
(Table 2). All pilots had normal or corrected-to-normal 
eyesight and were not color-blind. The study took about 
two hours, of which about one hour was spent in the 
simulator, and pilots were not paid for their participation. 


Table 2. Pilot Demographics 
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Approach Descriprion 

For each approach or run, the simulator w'as set to a 
previously saved checkpoint that positioned the helicopter 
at 250 feet above mean sea level and 2600 feet back of the 
stem of the ship. Wind and turbulence conditions that 
would produce a landing of difficulty 1-4 had been 
previously programmed into the simulator, and the 
appropriate hazard indicators were turned on at the 
beginning of the approach (if an indicator was supposed to 
be present). The simulator flight controls were trimmed to 
a 30-knot airspeed, and the pilots were given a verbal 
clearance to-land on one of four landing spots and the wind 
direction. The pilots were asked if they were ready, and 
then the simulator was set running. Pilots flew until the 
landing was complete, they verbally called out an aborted 
approach, or they crashed. Then the simulator was stopped 
and set up for the next run. Pilots were encouraged to make 
verbal comments during the test, and the entire test was 
videotaped for all pilots. The video camera was positioned 
behind the pilot, facing the projection screens, so that the 
pilot would not be visible on the tape. 

Approach Scenarios 

Scenarios were labeled based on which landing spot the 
pilot would be cleared for and where the airflow hazard 
would occur under certain wind conditions. 

Scenario A (“Aft”): Direct stem approach to landing 
spot 9, the aft-most landing spot on the LHA. With a direct 
bow wind, and at high wind speed and turbulence levels, 
an airflow hazard would occur downwind of the ship 
superstructure over landing spot 9. 

Scenario B (“Bow”): A 45-degree approach to the most 
forward spot on the bow of the ship, spot 1, and winds 
directly from the bow. This created an area of heavy 
downdraft (“suckdown”) directly over spot 1, which was 
often unexpected as it occurred even at relatively low 
winds and even in smooth wind conditions. 

Scenario P (“Port”): A 45-degree approach to the port 
side of the ship, to landing spot 7, just forward of the 
elevator and next to the ship superstructure. Winds from 
300 degrees (assuming the ship is moving toward the north 
or 360 degrees) caused a rotor to form over the deck edge 
just over lauding spot 7. Again, this hazard formed even at 
relatively low winds. 

Scenario S (“Starboard”): A 45-degree approach from 
starboard to landing spot 3A just forward of the ship 
superstructure. When winds are from 60 degrees, a vortex 
forms just at the deck edge and beside landing spot 3A. 



Landing Difficult} 7 Level 

We used four different landing difficulty levels (Table 
3) based on the Navy’s Pilot Rating Scale of landing 
difficulty [7], Each pilot flew each approach scenario at all 
landing difficulty levels. For each of LD 2 through 4, each 
pilot flew one approach with and one without a visual 
hazard indicator. For LD 1, each pilot flew one approach 
without a hazard indicator. Thus, each pilot flew 7 
approaches in each of the 4 landing scenarios, a total of 28 
approaches per pilot. The approaches were designed to 
take about 1-2 minutes each; therefore, the entire 
simulation took about one hour per pilot; this time length 
was designed to prevent pilot fatigue. 


Table 3. Landing Difficulty Levels 
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Landing difficult} 7 1 (LD i) - Control: These 
approaches showed how well the pilot could operate the 
simulator in the absence of particular hazards, and also 
provided periods of rest to the pilots to reduce fatigue and 
avoid discouragement (since the test consisted of an 
abnormally high percentage of very challenging landing 
conditions). 


Landing difficulty 2 (LD 2): Testing for negative 
effects of the hazard indicator. This difficulty level 
required moderate pilot effort. The hazard indicator (if 
present) was a translucent yellow object outlining the area 
where turbulent flow could be found. Because the 
conditions at LD 2 are considered to be within normal pilot 
abilities, we would expect few crashes even without the 
hazard indicator. The hypothesis tested at LD 2 was that 
the hazard indicator would not increase the crash rate (e.g. 
by distracting the pilot). Pilots were instructed that the 
yellow hazard represented caution and that they could 
continue the approach. 


Landing difficulty 3 (LD 3): Testing for benefit of 
hazard indicator. This difficulty level required maximum 
pilot effort. The hazard indicator was the same type as for 
the LD 2 approaches. Pilots were told that yellow 
represented caution and they were to continue the 
approach. A higher crash rate was expected at LD 3 
cornmensurate w T ith the more challenging conditions 
compared with LD 2. We hypothesized that the hazard 


indicator would reduce this crash rate - ideally, to a rate 
comparable to LD 2. 

Landing difficulty 4 (LD 4): Testing for benefit of 
hazard indicator with pilot instructional procedure. At LD 
4, safe landings were not probable. Fifteen pilots were told 
that if they detected a red hazard indicator along their 
approach path, standard operating procedure (SOP) was to 
abort the landing immediately. (The sixteenth pilot, who 
was not initially given this instruction, spontaneously 
proposed that it should be standard operating procedure.) 
These approaches test whether the same hazard indication 
methodology used for reducing the crash rate in marginal 
conditions will also operate reasonably in extreme 
conditions. 

Order of Presentation 

To compensate for possible learning effects, naif the 
pilots flew scenarios A and P without the hazard indicators 
and scenarios B and S with the hazard indicators during the 
first half of the test, and then conversely for the second 
half. The other pilots flew scenarios A and P with hazard 
indicators and scenarios B and S without indicators during 
the first half of the test. This was accomplished by defining 
an approach order randomly within these constraints, then 
reversing it to create a second order, then switching the 
first and second halves to create a third and fourth order. It 
was arranged that the most difficult approaches would not 
all follow one another, to reduce the likelihood of pilot 
fatigue. 


Dspcndsnt \7 jjrijiljlcs 

During the simulation, 50 variables, such as velocity 
and position of aircraft in x, y, z, control stick position both 
lateral and longitudinal, collective and pedal positions, 
landing gear forces, etc., were collected by the flight 
simulator at 1 0 Hz and stored in data files labeled for each 
run and pilot. From these data, we computed the crash rate. 
A “crash” was defined as an impact with the ship’s deck 
with a vertical velocity of 12 feet per second (fps) or 
greater as measured by the simulator. (This value is based 
on the US Navy standard structural limitation for 
helicopters. In order to be certified for shipboard use in the 
US Navy, rotorcraft must be able to withstand an impact of 
12 fps upon touchdown [23, 24].) 

We also gathered subjective pilot opinions from a 21- 
probe Likert-scale (1-5) questionnaire administered to the 
pilots at the end of the simulation. For each probe, the 
pilots had to circle one of “Strongly Disagree” (1), 
“Disagree” (2), “Neither Agree Nor Disagree” (3), “Agree” 
(4), and “Strongly Agree” (5). 



Hypotheses 

We tested five hypotheses: 

1. Crash rate will be reduced by the presence of 
hazard indicator (LD 3). 

2. Crashes will be eliminated by red hazard indicator 
if a standard operating procedure (SOP) is given to 
the pilots (LD 4). 

3. Hazard indicator will not cause distraction or 
degradation in performance in situations where 
adequate performance is expected without 
indicator (LD 2). 

4. Pilots will say they would use airflow' hazard 
visualization system. 

5. Pilot workload (as measured by frequency of 
control travel oscillation) will be reduced in the 
presence of the hazard indicator. 

Results 

In this section, we present the analysis of crash rate 
data, other flight statistics and subjective data, and 
illustrate the analysis with relevant pilot comments. Our 
hypotheses were generally confirmed by -the data. Pilot 
feedback was as a rule favorable to the system, and, 
additionally, indicated directions for further study. 

Hypothesis 1 confirmed. For the test at landing 
difficulty 3, there were 12 crashes out of 64 approaches 
wdthout the hazard indicator (crash rate .19, standard error 
.049) and 4 crashes out of 64 with the hazard indicator 
(crash rate .063, standard error .030) (Table 4). A t-test for 
paired samples show's that the hypothesis that the presence 
of the hazard indicator reduces the frequency of crashes 
during simulated shipboard helicopter landings is 
confirmed (t=2.39, df=63, p<0. 00985). 


Table 4. Landing Difficulty 3 - Crash Data 



. Landing Difficulty 3: 

Crash Rate vs. Presence of Hazard Indicator 


Hazard Indicator 


These strong results indicate the system should improve 
helicopter flight safety under hazardous conditions. 


Duong the tests, pilots remarked several times mat the 
indicators were helpful warnings; that they w'ere able to 
modify their flight path or power settings to counteract the 
known hazardous conditions, or make appropriate safety 
decisions based on knowledge gained from viewing the 
hazard indicators. Additionally, in the approaches without 
hazard indicators, pilots commented on several occasions 
that they were surprised by the wind conditions as they 
entered the hazardous areas, even though they had usually 
deduced that conditions were extreme before they entered 
the hazard zones. In a few of these runs w'here the pilot 
made such a comment, the approach terminated in a crash. 

Hypothesis 2 confirmed. At landing difficulty 4 
(beyond the capacity of the aircraft), there were 0 crashes 
in 64 approaches w'ith the hazard indicator as opposed to 
15 crashes out of 64 without the indicator, for crash rates 
of 0% and 23% respectively. (Standard errors were 0 and 
.053.) A t-test for paired samples shows that this 
hypothesis — that the presence of the red hazard indicator 
combined with appropriate instructions to the pilot 
prevents crashes — is strongly confirmed (t=4.39, df=63, 
p<000022). What this means is that although pilots may 
sometimes continue into a situation that is beyond the 
capacity of the aircraft if they do not have sufficient 
knowledge of the danger of the situation, giving them the 
appropriate information in a clear and simple manner 
during the approach can prevent accidents. This is an 
improvement over the current envelope system because, as 
one pilot noted, it would be very helpful in case the w'inds 
shifted during the approach. If he suddenly saw a red 
hazard area appear on deck, he would know 7 immediately to 
abort the approach. 

Hypothesis 3. Mo negative effect of liazard. indicator. 

It appears that the hazard indicators did not distract the 
pilots. The crash rate at LD 2 w'as the same with and 
without the mdicator. Crash rate for both was identical, 
7.8% or 5 crashes out of 64 for each set of approaches. 
(Standard error was .034.) However, because the crash rate 
w'as low', with a sample of this size it is not possible to 
conclusively state that the hazard indicator made no 
difference in crash rate. On the other hand, the pilots did 
not feel the hazard indicators were distracting. On our 
simulation evaluation questionnaire, probe 6 was, “The 
airflow' hazard visualization distracted me from the task of 
flying the aircraft.” The pilots disagreed with this 
statement: 94% of the pilots answered “Strongly Disagree” 
(1) or “Disagree” (2) with the median “Disagree” (2). 

Hypothesis 4 confirmed. When pilots were asked to 
report their level of agreement with the statement, “I would 
use this system if it W'ere available on my aircraft,” eight 
pilots chose “Strongly Agree” (5), five chose “Agree” (4), 
one chose “"Neither Agree Nor Disagree” (3) and two chose 
“Disagree” (2). Median response was 4.5, between 
“Strongly Agree” and “Agree” (Table 5). This indicates 



confirmation of Hypothesis 4, that pilots would use the 
system. 


Table 5. Probe 21 Results 



21. I would use this display system if it were available on my 
aircraft. 


: Agree:;; 


.... . .. .. .. ... ... . . ... - 

; Neither Agree * ' ' 

Nor Disagree 

81% Agree. 13% Disagree 
Median 4.5. Sta Dev 1.0 


Hypothesis 5. Previous work has indicated that 
rotorcraft pilot workload may be estimated by frequency of 
control oscillation, specifically lateral cyclic movements 
[5], We applied Fourier analysis to the time-series data of 
lateral cyclic position (XA) obtained from the simulator, 
initially with specific emphasis on those runs which 
terminated in a crash. 

The following graph (Table 6) illustrates those results 
for one pilot during three different approaches to the 
landing spot aft of the island: a control run at landing 
difficulty 1, a run at landing difficulty 3 without the hazard 
indicator, and a run at landing difficulty 3 with the 
indicator. The run at LD 3 with no hazard indicator 
terminated in a crash; the other two were completed with a 
successful touchdown. We applied the Fourier transform 
to the last 25.6 seconds of each of the time senes landing 
runs, since the final critical seconds before touchdown are 
the most indicative of maximum pilot workload. The 
spectrum was analyzed in a range of frequencies around 
1 Hz, where control movements can be presumed to be 
intentional (rather than, say, due to aircraft vibration). 
Low frequency movements (at approximately 0.5 Hz and 
below) are indicative of normal control travel, whereas 
previous work has suggested that frequency peaks near 
1 Hz indicate a dangerously overloaded pilot [5], 


Table 6. Typical lateral cyclic power spectrum, last 
25.6 seconds of approach to landing spot aft of island 
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Table 6 shows the Fourier power spectrum for XA, the 
lateral cyclic position. The Fourier power coefficients P, in 
the plot are obtained as the squared magnitudes of the 
complex-valued Fourier amplitudes, 

P, = |X ; | 2 = Re(X,(XA)) 2 + fm(X,(XA)) 2 

where X ; is . the i-th component of the Fourier transform of 
the XA time series. 

For the run at LD 3 with no indicator, stronger peaks 
occur at higher frequencies than the other two runs. Peaks 
at approximately .6, .7, and .9 Hz are visible, with lesser 
peaks at 1.1 and 1.2 Hz; there are no corresponding 
frequency peaks for the other two runs. The total spectral 
energy' in this range near 1 Hz is about twice as great in the 
run at LD 3 without a hazard indicator, as compared to the 
run with a hazard indicator or the control run at LD 1 . Data 
analysis is ongoing in this area, but this preliminary 
analysis of lateral cyclic travel appears to point to a 
reduction in pilot workload with the presence of the hazard 
visualization system. 

Control group (LD 1). Because conditions in the 
simulator are somewhat different than in a real helicopter, 
and visual and proprioceptic feedback is reduced (no chin 
bubble through which helicopter pilots can look down past 
their feet and see how close they are to the deck, no depth 
perception in the visuals, no bump when the landing gear 
contacts the deck, etc.), and especially because pilots are 
flying it for the first time without any training with an 
instructor (the usual procedure when transitioning to a new 
aircraft), a certain number of crashes in the simulator are to 
be expected. For this reason we included a set of low- 
hazard approaches in the study to serve as a control (LD 1). 

The crash rate at landing difficulty 1 was 9.4% (6 out of 
64, standard error .037), which is not significantly different 
from LD 2 or LD 3’s crash rates (5 out of 64, std. err. .034 
and 4 out of 64, std. err. .030, respectively; t-test, p=0.38 







and p—0.26) when the hazard indicator is present. In other 
words, the use of the hazard visualization system reduced 
the crash rate to the same level as that of the control 
approaches. 

Summary of Crash Statistics 

This subsection describes the overall crash statistics for 
our experiment, where, as explained earlier, a “crash” was 
defined as an impact with the ship’s deck of more than 12 
feet per second. (Table 7) summarizes all the data, and the 
following sections describe further statistical analysis of 
the data and our interpretations. 


Table 7. Crash Statistics for All Landing Difficulties 


Landing 

Difficulty 

Hazard. 

Indicator 

Crashes 

Total 

Approaches 

Crash 

Rate 

Standard 

Error 

LD 1 

No 

6 

64 

0.0938 

0.0367 


No 

5 

64 

0.0781 

0.0338 


Yellow 

5 

64 

0.0781 

0.0338 


No 

12 

64 

0.188 

0.0492 


Yellow 

4 

64 

0.0625 

0.0305 


No 

15 

64 

0.234 

0.0534 

LD4 

Red 

0 

64 

0 

0 


Learning effects 

For the first half of the simulator test, the pilots crashed 
25 times out of 224 approaches flowm for a crash rate of 
11.2%, while in the second half of their tests, the pilots 
crashed 22 times out of 224 approaches, for a crash rate of 
9.8% (Table 8). 


Table 8. No apparent learning effects in study 



This is not a significant difference (t-test, t=0.46, 
df=445, p=0.32), although the pilots did state that they 
believed they performed better as they flew the simulator 
longer. (Probe 17: “It became easier over time to fly 
because my experience on the simulator increased.” Eight 


pilots answered “Strongly Agree” (5), six pilots chose 
“Agree” (4), and two pilots chose “Neither Agree or 
Disagree” (3). Median response was 4.5.) This appears to 
indicate that learning effects did not bias our study, as was 
intended in its construction. 

We considered whether there could be other reasons for 
this result. For example, a few of the pilots commented 
towurd the end of the 28 approaches that they were getting 
tired. It is possible that there were more crashes at the 
beginning of the flights, but this effect was masked by 
more crashes at the very end of each pilot’s simulator time. 
In order to test this theory, we graphed crashes as a 
function of approach order (Table 9). In this graph, the x- 
axis lists the order flown, from 1 to 28, and the y-axis the 
number of crashes at that point (out of 16 approaches 
flown). The graph makes evident that there is no such 
pattern of bias in the number of crashes as a function of 
approach order. Therefore, we concluded that learning 
effects did not bias our study. 


Table 9. No evidence for learning effects or other 
global effects as a function of order flown 



Waveoff Rate Analysis 

In order to address the concern that the hazard 
indicators may simply make a pilot more cautious, and that 
the reduction in crash rate was solely due to an increase in 
aborted landings, we also analyzed the overall waveoff 
rates. A waveoff is an aborted landing, or “go-around,” 
where the pilot decides a safe landing is not probable, and 
proceeds to climb to re-enter the pattern and (possibly) 
attempt the landing again. In our experiment, as soon as 
the pilot called for an aborted landing, we terminated the 
run, and the pilot did not attempt another landing under 
those conditions. 

In reality, were a pilot to wave off, the next step would 
most likely be another landing approach, perhaps calling 
for the ship to turn further into the wind, or perhaps 



requesting a different landing spot. However, for the 
purposes of our simulation, we counted waveoffs 
separately from completed landings. Each approach, 
therefore, took one of three possible terminations: a 
completed landing, a waveoff, or a crash. Because go- 
arounds are a frequent and necessary part of safe flying, for 
our main analysis above we considered the crash rate as 
our primary dependent variable in determining whether or 
not our system had a positive effect on flight safety under 
the stated conditions. 

Many flight instructors believe that students should be 
taught that all landing approaches should really be 
considered approaches to go around. .Any number of go- 
arounds are better than making a destabilized approach to 
landing that could end in a crash. Because this attitude is 
common in the aviation community, an increased number 
of go-arounds would not be considered a negative result. 
However, it can be supposed that there are operational 
considerations in naval aviation whereby a waveoff is 
costly in some sense (although it preserves the aircraft and 
pilot). Therefore, a hazard indication system that does not 
increase waveoffs would be (other factors equal) preferable 
to one that’ does. Waveoff data is summarized in (Table 
10 ). 


Table 10. Go-Around Statistics for Ail Landing 
Difficulties 


Landing 

Difficulty 

Hazard 
- Indicator 

Go- ' 
Arounds 

Total 

Approaches 

'Go-Around 

Rate 

Standard 
Error I Y 

LD1 

No 

3 

64 

0.0469 

0.0266 

LD 2 

No 

17 

64 

0.266 

0.0556 


12 

64 


0.0492 

LD 3 

No 

22 

64 

0.344 

0.0598 

Y eilow 

23 

64 

0.359 

0.0605 


The waveoff rate at landing difficulty 2 with no hazard 
indicator was 17 out of 64 approaches (a rate of 0.266 with 
a standard error of 0.0556) and 12 out of 64 (a rate of 
0.188 with a standard error of 0.0492) with the hazard 
indicator present (Table 11). This is not a significant 
difference (t=1.04, df=63, p=0.15) for landing difficulty 2. 


Table 1 1 . Landing Difficulty 2 - Go-Around Data 



At landing difficulty 3, the waveoff rate when the 
hazard indicator was absent was 22 out of 64 (a rate of 
0.344 with a standard error of 0.0598), almost identical to 
the rate when the hazard indicator was visible, 23 out of 64 
(a rate of 0.359 with a standard error of 0.0604), again, not 
a significant difference (t=0.18, df=63, p=0.427) (Table 
12 )/ 


Table 12. Landing Difficulty 3 - Go-Around Data 



For cases where the ANOVA statistical test is 
applicable, it is a more conservative test of significance 
than individual t-tests. For the waveoff data, a two-way 
ANOVA on landing difficulty (2, 3) and hazard indicator 
(present absent) shows neither a significant difference due 
to either factor alone, nor a significant interaction between 
the factors (Fcrit = 6.7; for landing difficulty F = 4.9, 
p=0.028; for hazard F= 0.31, p=0.58; for the interaction 
F=0.69, p=0.41). 


We did not analyze the data for landing difficulty 4 
because we instructed fifteen of the sixteen subjects to 
wave off whenever they detected a red hazard indicator in 








their path, so any results from landing difficulty 4 would be 
artificial. 


Table 14. No significant difference between pilot 
groups based on experience level 


It appears, therefore, that the presence or absence of the 
hazard indicator at landing difficulties 2 and 3 does not 
affect the waveoff rate. Thus, analyzing the waveoff data 
does not lead to any changes in our conclusions about the 
four hypotheses described above. 


Analysis by Pilot Experience Level 

An interesting question was whether pilot experience 
level had any effect on performance, and on the 
effectiveness of the hazard indicators. In order to look at 
this question, we divided the 16 pilots into three groups, 
where there were natural gaps in their experience levels: 
less experienced, moderately experienced, and highly 
experienced (Table 13). 


Table 13. Pilots grouped by experience level 


Pilot Experience Level 

Helicopter Flight Hours 

Number of Pilots in 
Group 

Less experienced 

200 - 850 

5 

Moderately 

experienced 

1500-3200 

7 

Highly experienced 

4000 - 7300 

• 

4 


Crash Rate vs. Pilot Experience Level 



Pilot Experience Level 


■ LD 2/Nc 

■ I'D 2/Haz 
E LD 3/fvc : 
;GLD 3/Haz 

■ LD 4/No | 

■ LD 4/f-.az 


In other words, the trends suggest that our hypotheses 
1, 2 and 3 are true for the group of highly experienced 
pilots as well. 


Analysis of Subjective Data from Pilot Evaluations 

All pilots filled out a 21 -probe Likert-scale post- 
simulation evaluation. The possible responses were (1) 
Strongly Disagree, (2) Disagree, (3) Neither Agree Nor 
Disagree, (4) Agree, or (5) Strongly Agree. In this section, 
■we present the results of some of the probes other than 
those previously discussed in this chapter. 


One of the very experienced pilots had commented that 
he did not leam anything new from the placement of the 
hazard indicators, but he felt it might be a good training aid 
for more inexperienced pilots. Additionally, most of the 
less experienced pilots stated that they did leam something 
from the hazard indicators. 

We therefore examined the data for evidence that the 
decrease in crash rates was concentrated among the pilots 
with less experience.. The reduction, however, was seen 
across all experience levels, although we could not obtain 
statistical significance in most cases due to the lower 
sample numbers. The data is summarized below (Table 
14). 


Probe 4. I would be more cautious if I saw a yellow 
airflow hazard in my approach path. 


As (Table 15) illustrates, pilots exhibited caution upon 
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commented that they changed their flight paths based on 
the location, of the hazard indicators. We conjecture that 
this pilot action contributed to the lower crash rates at 
landing difficulty 3 when the yellow hazard indicators were 
present. One pilot did warn of the possibility that the 
hazard indicator could make pilots overcautious; however, 
the waveoff data did not seem to bear this out (there was no 
increase in waveoffs with the presence of a yellow hazard 
indicator). 



Table 15. Probe 4 Results 



I would be more cautious if I saw a yellow airflow hazard in my 
approach-path., 

;■ . 


Strongly Agree" 


(Neither 'Agree i Nor 
Disa^^iV:- • 
- 'Pilot response ' 


'Aoree: 

Tr." 


100% Agree, 0% Disagree 
Median 5, Std Dev 0.45 


Probe 11. The shape of the airflow hazard ions overly 
simplistic and did not present enough information. 

Most of the pilots disagreed with this statement (Table 
16). However, the bimodal distribution of responses 
coincides with pilot post-simulation commentary: it seemed 
that the pilots fell into two groups, one that wanted more 
information on the indicators, perhaps even some 
animation, and another that felt “the simpler, the better.” A 
few pilots commented that they wanted a quantitative value 
for airflow speed as well as the qualitative indication of 
whether the hazard was beyond aircraft limits. 


low-fidelity prototype, in this study we did not show them 
any moving indicators. The strong, almost visceral 
reaction of the pilots in the earlier study always occurred as. 
they were viewing an animated indicator on the screen. 
Additionally, when a few of the pilots who agreed with this 
probe statement were queried as to the type of motion, they 
concurred that the animation should not be too rapid, and 
all of them wanted the ability to stop the animation, 
especially close to the end of the approach. 


Table 17. Probe 13 Results 



Table 16. Probe 11 Results 



11. The sha^e of the airfloiy hazard was overly simoiisfic and did 


38% Agree, 56% Disagree 
.. Median. 2, Std Dev 1.2 


Probe 14. It would be distracting if the hazard indicator 
showed airflow motion. 


Although the pilots mostly disagreed with this 
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When the pilots who wanted airflow motion were asked for 
a reason, many stated that they wanted more information 
about the hazard at the beginning of the approach. Just as 
with probe 13, they concurred that they wanted to be able 
to turn off any motion. 


Probe 13. It would be helpful if the hazard indicator 
moved to display airflow motion. 

(Table 17) illustrates the spread of opinions on 
indicator motion. Although the pilots were not as negative 
about motion or animation in this study as they were in the 




Table 18. Probe 14 Results 



Probes 13 and 14 together indicate a need for further 
study on the use of animated indicators, as the benefits 
evidently anticipated by the pilots in the simulation shady 
do not jibe with the strong aversion expressed by pilots in 
the low-fidelity prototype study. 

Probe IS. The presence of the hazard indicators gave me 
more confidence as to the state of the winds and airwake 
on deck. 

The pilots were almost unanimously in agreement with 
this statement (Table 19). The only pilot who disagreed 
was one of the most experienced pilots in our group, who 
stated that he already knew where all the hazardous areas 
were. We discuss this pilot’s opinions further in the final 
section on pilot comments sncl suggestions 


Table 19. Probe 18 Results 



Probe 19. I learned something about the location and 
effect of hazardous airwake on the deck of a ship by 
seeing the hazard indicators. 

Again, the pilots agreed with this statement (Table 20). 
The same experienced pilot that disagreed with probe 18 
disagreed here; he said he already knew all about the 
location of hazardous airwake on ships. Indeed, he was 
one of the few pilots who did not crash at all during the 
simulation. The two who were neutral on this question 
were also relatively experienced. 


Table 20. Probe 19 Results 



Probe 21. I would use this display system if it were 
available on my aircraft 

We have previously discussed this result (see Table 5) 
as it directly relates to our Hypothesis 4. Overall, the fact 
that 81% of the participating pilots said they would use the 
system is a strong indication that such a system would meet 
with pilot acceptance if it were implemented and installed 
in aircraft. Combined with the strong positive results in 
pilot performance in the simulation study, it is evident that 
further research into the implementation of an in-cockpit 
airflow hazard display system is called for. 

Pilot comments and suggestions 

At the end of our questionnaire, probe 22 was an open- 
ended request for comments. We also gathered verbal 
commentary and suggestions from the pilots during the 
post-flight debrief. Several pilots commented extensively. 





In this section, we give some of their responses and 
suggestions. 

As discussed earlier, there appeared to be a bimodal 
distribution of pilot opinions on whether the indicators 
were overly simplistic and needed to provide more 
information, or that more information would be distracting. 
We present quotes from two pilots who illustrate the 
opposing viewpoints: 

One of the most experienced pilots in our study (who, 
however, did not have any helicopter shipboard landings) 
commented, “Interesting concept - needs some better 
depiction of what the hazard really is, i.e. vortex, rooster 
tail. Some velocity information would give the pilot some 
valuable lead information to anticipate what to do.” 

On the other hand, a pilot with a moderate amount of 
experience but with many helicopter shipboard landings, 
said, “with all you have to do, landing... controlling your 
deceleration] . .. especially at night... you don’t want any 
distraction” in the form of animation or numeric 
indications in the hazard visualization. 

These comments indicate that further studies are called 
for, where different types of hazard indicators, some with 
an indication of airflow motion, some animated, some with 
numeric readouts, are compared. 


suggest the potential benefits of integrating visual hazard 
indicators with out-the -window views or synthetic vision 
systems. 

Many pilots spontaneously mentioned helicopter 
accidents that they felt could have been avoided if the 
pilots had had access to a system like this one. One pilot 
mentioned the Mount Hood Pave Hawk crash in 2002, 
where a helicopter in the process of rescuing nine hikers 
napped in a crevasse on a mountaintop suddenly crashed 
[26]. The weather was clear and sunny, but there were 
gusty winds, as is typical around a mountaintop. This pilot 
believed that unseen turbulence and/or downdrafts beyond 
the capability of the helicopter were the likely causes of the 
crash. 

Another commented that in his work as a medevac 
pilot, he hated landing on top of Stanford Hospital, 
“especially at night.” “There’s always a vortex there,” he 
said. 

One pilot had a relative who flew helicopters in 
firefighting. Backdrafts and up- and downdrafts cause 
tremendous dangers for firefighting pilots. A system like 
this “could really make a difference,” according to this 
pilot. 

Conclusions and Further Work 


Another area for further research lies in making the 
display adaptive. Several pilots commented that they 
wanted more detail at the beginning of the approach and 
less at the end. To that end, perhaps an adaptive display 
might be successful. The display could adapt based on 
where the pilot was in the approach, or could be more 
sophisticated and track pilot workload through 
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that could be selected by the pilot. 


One pilot said he would prefer a hazard indicator that 
was not in the visual field. Another stated that night 
operations were more important than day VFR (Visual 
Flight Rules), and that the indicating system must be 
studied at night for it to be useful. Night operations would 
be another fertile area for future research. 


Based on the results of a flight simulation usability 
study, we believe that simple, real-time visualization of 
airflow may improve helicopter pilot landing performance. 
The use of a simple, static visualization of airflow hazard 
location and severity leads to a significant decrease in crash 
rate for a critical class of landings /those where landing is 
permitted, but difficult). It also appears that the visual 
system does not distract the pilots nor cause degradation in 
their landing performance. Power spectrum analysis of 
lateral cyclic position during the simulated landing runs 
also points to a reduction in pilot workload with the hazard 
visualization system. Additionally, pilot feedback was 
generally favorable to the system. Further studies are 
called for to verify these results. 
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